Introduction: What Is Statistical Learning Theory?

Let us start things off with a simple illustrative example. Suppose someone hands you a coin that has an unknown probability θ of coming up heads. You wish to determine this probability (coin bias) as accurately as possible by means of experimentation. Experimentation in this case amounts to repeatedly tossing the coin (this assumes, of course, that the bias of the coin on subsequent tosses does not change, but let’s say you have no reason to believe otherwise). Let us denote the two possible outcomes of a single toss by 1 (for HEADS) and 0 (for TAILS). Thus, if you toss the coin n times, then you can record the outcomes as X1, . . . , Xn , where each Xi ∈ {0,1} and P(Xi = 1) = θ independently of all other X j ’s. More succinctly, we can write our sequence of outcomes as X n ∈ {0,1}n , which is a random binary n-tuple. This is our sample. What would be a reasonable estimate of θ? Well, by the Law of Large Numbers we know that, in a long sequence of independent coin tosses, the relative frequency of heads will eventually approach the true coin bias with high probability. So, without further ado you go ahead and estimate θ by